Image processing apparatus and method of controlling the same

ABSTRACT

An image processing apparatus that can suppress a tint phenomenon in image recovery processing and a method of controlling the same are disclosed. A saturation determination unit determines whether or not a saturated pixel is included in a reference region referred to by filter processing units. An R component output adjustment unit and a B component output adjustment unit adjust corresponding color components resulting from image recovery processing according to the determination result obtained by the saturation determination unit.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and amethod of controlling the same, and in particular relates to an imagecorrection technique.

2. Description of the Related Art

In order to correct image degradation that occurred due to an aberrationor diffraction phenomenon in an imaging optical system, processing(image recovery processing) is known in which degradation in an image iscorrected using information regarding the optical transfer function(OTF) of the imaging optical system (refer to paragraphs 0008 to 0013 ofJapanese Patent Laid-Open No. 2013-51599). Also, letting g(x,y) be thedegraded image and f(x,y) be the original image, R(x,y) in the followingequation is called an image recovery filter.

g(x,y)*R(x,y)=f(x,y)

Note that the operator “*” in the above equation represents convolution(a product-sum operation), and (x,y) represents coordinates in theimage.

Letting the optical transfer function of the imaging optical system beH(u,v), the image recovery filter is obtained by calculating the inverseFourier transform of 1/H. Note that (u,v) represents coordinates in atwo-dimensional frequency surface, that is to say, represents afrequency. The image recovery filter that is applied to atwo-dimensional image is generally a two-dimensional filter that has atap (cell) corresponding to each pixel in the image.

The extent of image degradation occurring due to aberration or lightdiffraction in the imaging optical system varies since the influence ofaberration and diffraction differs depending on the wavelength ofincident light, that is to say differs for each color component, andtherefore recovery filters having different characteristics are appliedfor each color component. Generally, with an image acquired by animaging element having a pixel size of approximately several μm, theinfluence of aberration and diffraction extends to several tens ofpixels, and therefore the recovery filter needs to be a filter that hasmany taps in order to be able to refer to a wide range of pixels.

With filter processing that refers to a wide range of pixels, imagequality is readily negatively influenced due to the inability to obtaina correct signal value in and around a saturated portion of the inputimage. With image recovery processing that applies filters havingdifferent characteristics for each color component, tinting readilyoccurs due to a loss of balance between the color components.

As one example of a method of suppressing such effects, Japanese PatentNo. 4599398 discloses a technique for suppressing the effect of filterprocessing according to the brightness of the image area being referredto by the filter.

However, the conventional techniques disclosed in Japanese PatentLaid-Open No. 2013-51599 and Japanese Patent No. 4599398 do not giveconsideration to maintaining balance in output values among filters. Forthis reason, it has not been possible to address a problem such astinting, which occurs in processing that applies separate filters foreach component of an image signal as in image recovery processing.

SUMMARY OF THE INVENTION

An object of the present invention is to provide an image processingapparatus that can suppress a tint phenomenon in image recoveryprocessing and a method of controlling the same.

According to an aspect of the present invention, there is provided animage processing apparatus comprising: a filter unit configured toperforming image recovery processing by applying a spatial filter foreach of color components of an image; a determination unit configured todetermining whether or not a saturated pixel is included in a referenceregion that is a region of pixels referred to by the spatial filter; andan adjustment unit configured to adjusting a color component among thecolor components resulting from the image recovery processing accordingto a determination result obtained by the determination unit.

According to another aspect of the present invention, there is providedan image processing apparatus comprising: a filter unit configured toperforming image recovery processing by applying a spatial filter foreach of color components of an image; a determination unit configured todetermining whether or not a pixel whose value is greater than or equalto a predetermined threshold value is included in a reference regionthat is a region of pixels referred to by the spatial filters; and anadjustment unit configured to adjusting a color component among thecolor components resulting from the image recovery processing accordingto a determination result obtained by the determination unit.

According to a further aspect of the present invention, there isprovided a method of controlling an image processing apparatus,comprising: a filter step of performing image recovery processing byapplying a spatial filter for each of color components of an image; adetermination step of determining whether or not a saturated pixel isincluded in a reference region that is a region of pixels referred to bythe spatial filters; and an adjustment step of adjusting a colorcomponent among the color components resulting from the image recoveryprocessing according to a determination result obtained in thedetermination step.

According to yet further aspect of the present invention, there isprovided a method of controlling an image processing apparatus,comprising: a filter step of performing image recovery processing byapplying a spatial filter for each of color components of an image; adetermination step of determining whether or not a pixel whose value isgreater than or equal to a predetermined threshold value is included ina reference region that is a region of pixels referred to by the spatialfilters; and an adjustment step of adjusting a color component among thecolor components resulting from the image recovery processing accordingto a determination result obtained in the determination step.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically showing the overall flow of imageprocessing according to an embodiment.

FIG. 2 is a block diagram showing an example of a functionalconfiguration of an image processing apparatus according to anembodiment.

FIG. 3 is a flowchart for describing a color difference adjustment valuegeneration method used in an embodiment.

FIGS. 4A to 4C are diagrams for describing interpolation processingaccording to an embodiment.

FIGS. 5A to 5C are diagrams for describing effects obtained by a firstembodiment.

FIGS. 6A to 6D are diagrams for describing effects obtained by a secondembodiment.

FIG. 7 is a block diagram showing an example of a basic functionalconfiguration of an image processing apparatus that performs imagerecovery processing.

FIGS. 8A to 8D are diagrams for describing an overview of filterprocessing in image recovery processing.

FIGS. 9A to 9D are diagrams for describing a tint phenomenon that occursin image recovery processing.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will now be described indetail in accordance with the accompanying drawings. First, basic imagerecovery processing and problems that occur therein will be described.FIG. 7 is a block diagram showing an example of the basic functionalconfiguration of an image processing apparatus 400 that performs imagerecovery processing.

An image data acquisition unit 401 reads out, from image data targetedfor processing, image data that corresponds to a partial region as shownby 501 in FIG. 8A. This partial region is a region made up of pixelsthat are to be referred to in one instance of image recovery processingperformed by later-stage first to fourth filter processing units 402 to405, and therefore will be called a “reference region” hereinafter.

Here, it is assumed that, for example, the first to fourth filterprocessing units 402 to 405 are each a 3×3 two-dimensional spatialfilter as shown in FIG. 8B (i.e., a filter that refers to 3×3×4 imagedata in one instance of image recovery processing). In this case, thereference region 501 is 6×6 pixels in size as shown in FIG. 8C. Afterreading out the image data corresponding to the reference region 501,the image data acquisition unit 401 separates it into color componentsG1, G2, R, and B as shown in FIG. 8D, and outputs the color componentsto the corresponding first to fourth filter processing units 402 to 405.Here, pixels 701 to 704 that are located at the center of the image dataseparated into the respective colors (and that correspond topre-separation pixels 601 to 604 in FIG. 8C) are the pixels that aretargeted for image recovery processing, and will be called “pixels ofinterest”. In other words, four pixels making up one unit of the Bayerarrangement are processed in one instance of image recovery processing.

The four first to fourth filter processing units 402 to 405 provided onefor each color component are recovery filters that perform imagerecovery processing on input image data of one color component. Sincethe influence of aberration and light diffraction differs depending onshooting conditions such as the aperture and focal length of the imagingoptical system, the values of the 3×3 filter coefficients of the firstto fourth filter processing units 402 to 405 are set according to theshooting conditions. The spatial filter processing is performed byperforming a product-sum operation on the color component values at the3×3 pixel positions and the corresponding filter coefficients. Forexample, in the recovery filter processing performed on the pixel 701 inFIG. 8D, the product of the color component value of the pixel 701 and afilter coefficient K22 is obtained, and in this way, the products offilter coefficients K11 to K33 and the values at their correspondingpositions are obtained, and the sum of the products is the processingresult. Note that although the example of a 3×3 two-dimensional spatialfilter is described here in order to simplify the description, it isactually desirable to use a two-dimensional spatial filter that has acombination of 30×30 coefficients or more.

Returning to FIG. 7, image data R′, G1′, G2′, and B′ that is the resultof the image recovery processing and output by the first to fourthfilter processing units 402 to 405 is then reverted to the Bayerarrangement shown in FIG. 8C and output by a recovery data output unit406.

The reading out of image data in a reference region, the execution ofimage recovery processing, and the output of processing results arerepeated such that all of the pixels in the image are subjected to theabove-described processing as pixels of interest.

The following describes a tint phenomenon that occurs in image recoveryprocessing. FIG. 9A shows an example of an image that includes anachromatic color edge, and FIG. 9B is a diagram in which the pixelvalues of a pixel line corresponding to the edge portion of the image inFIG. 9A are plotted in the X direction separately for each colorcomponent. In FIG. 9B, the horizontal axis indicates the pixel positionin the X direction, the vertical axis indicates the color componentvalue, and Th is a value indicating the saturated level for the colorcomponent values in the imaging element that generated the image datatargeted for processing.

FIG. 9B shows G component pixel values and R component pixel valuesprior to image recovery processing. These values are values after theexecution of white balance processing (gain adjustment givingconsideration to the spectral sensitivity of the imaging element and thecolor temperature of the light source) on the color component valuesread out from the imaging element. Note that here, before white balanceprocessing, the G component is saturated but the R component is notsaturated, and it is assumed that due to white balance processing, the Gcomponent value does not change, whereas the R component value isamplified. In the case of the G component, the values at pixel positionsX1 and X2 are saturated before white balance processing, and it is shownthat the original values are lost after white balance processing aswell. On the other hand, in the case of the R component, the values arebelow the saturated level before white balance processing, and thereforethey are amplified by an amount corresponding to the amount of gainadjustment in white balance processing. As a result, in the case of theR component, the values at the pixel positions X1 and X2 are valueshigher than the saturated component. Note that it is assumed that the Bcomponent pixel values (not shown) are similar to the R component pixelvalues.

It is assumed that the pixels in FIG. 9B are subjected to image recoveryprocessing by applying edge enhancement filters having differentcharacteristics for each color signal as shown in FIG. 9C in order tomake an improvement in terms of the reduction in contrast due todiffraction. Note that one-dimensional filters are applied in the Xdirection in this description in order to simplify the description andfacilitate understanding.

FIG. 9D shows component values resulting from the image recoveryprocessing. In the case of the R component (and the B component), filterprocessing can be carried out on all of the pixels using only theunsaturated component values, and therefore the original effects of edgeenhancement can be obtained. On the other hand, in the case of the Gcomponent, the saturated pixel X2 (or the saturated pixels X2 and X1) isreferred to in the filter processing in which the pixel position X3 isthe pixel of interest, and therefore the value resulting from the filterprocessing is not the correct value. In this example, the G component isgreater than the R component (and the B component) at the pixel positionX3 after the image recovery processing, and as a result, green tintingis observed in the vicinity of the pixel position X3.

On the other hand, the R component (and the B component) is greater thanthe G component at the pixel positions X1 and X2, but the valuesexceeding the saturated level (Th) will be reduced to the saturatedlevel through clip processing in later-stage development processing.These color components are therefore reduced to the saturated level anddisplayed as achromatic colors (blown-out highlights).

In this way, correct filter results are not obtained if saturated pixelsare referred to in image recovery processing that employs filterprocessing, and tinting occurs in the region in and around a saturatedportion. In particular, in the achromatic color edge region given as anexample here, the G component for which the imaging element has a highspectral sensitivity is more likely to be saturated than the R and Bcomponents, and therefore there is a tendency for the recovery amountfor the G component to be lower than the R and B components, and forgreen tinting to more readily occur as a result.

First Embodiment

The following describes an illustrative embodiment of the presentinvention.

FIG. 1 schematically shows the overall flow of image processing thatincludes image recovery processing. Image processing performed on RAWdata will be described here. It is assumed in the present embodiment aswell that the RAW data has the Bayer arrangement shown in FIG. 8C.

First, gain adjustment processing (white balance (WB) processing) 201 isapplied to input image data (RAW data) separately for each colorcomponent giving consideration to the color temperature of the lightsource and the spectral sensitivity of the sensor.

Next, image recovery processing 202 is applied to the image dataresulting from the gain adjustment. Details of the image recoveryprocessing will be described later. Similarly to the input image data,the output after the image recovery processing has the Bayerarrangement.

Next, development processing 203 such as color interpolation (demosaic)processing, noise reduction and sharpness processing, and gammaprocessing is applied to the image data resulting from the imagerecovery processing. YUV conversion processing 204 for conversion fromthe RGB format into the YUV format is then applied to the image dataresulting from the development processing, and thus this series of imageprocessing is completed.

Note that in the present embodiment, it is envisioned that image data ispassed between the function blocks and steps via a memory. Note that ifthe processing of the function blocks and steps is executed usingdedicated hardware modules, the data may be passed directly between thehardware modules. This is of course based on the presumption that thehardware modules have a sufficient buffer memory capacity for holdingthe data.

FIG. 2 is a block diagram showing an example of the functionalconfiguration of an image processing apparatus 100 according to thisembodiment of the present invention for executing the image recoveryprocessing 202 shown in FIG. 1.

Similarly to the image data acquisition unit 401 in FIG. 7, an imagedata acquisition unit 101 reads out image data corresponding to areference region whose size corresponds to the size of the spatialfilters used by first to fourth filter processing units 102 to 105. Theimage data acquisition unit 101 then outputs the image datacorresponding to the reference region as it is to a saturationdetermination unit 106, and also separates it into color components andoutputs them to the first to fourth filter processing units 102 to 105.

The saturation determination unit 106 refers to the image datacorresponding to the reference region, determines whether or not itincludes even one pixel whose value is greater than or equal to apre-set threshold value (saturated level), and outputs the result of thedetermination to an R component output adjustment unit 113 and a Bcomponent output adjustment unit 114. If the reference region includes asaturated pixel, it is determined that a recovery filter will refer to asaturated pixel in image recovery processing applied to the pixel ofinterest.

As described with reference to FIG. 7, the first to fourth filterprocessing units 102 to 105 perform image recovery processing for therespective color components and respectively output color componentvalues R′, G1′, G2′, and B′ resulting from image recovery processing.

In accordance with the result of the determination made by thesaturation determination unit 106, the R component output adjustmentunit 113 outputs either the output value R′ from the third filterprocessing unit 104 or a later-described tint correction value (colordifference adjustment value R″) to a recovery data output unit 115.Similarly, in accordance with the result of the determination made bythe saturation determination unit 106, the B component output adjustmentunit 114 outputs either the output value B′ from the fourth filterprocessing unit 105 or a later-described tint correction value (colordifference adjustment value B″) to the recovery data output unit 115.

Similarly to the recovery data output unit 406 in FIG. 7, the recoverydata output unit 115 then reverts the image data resulting from theimage recovery processing, which is input separately for each colorcomponent, to the Bayer arrangement and outputs it.

Next, the method of generating the color difference adjustment values R″and B″ will be described with reference to the flowchart in FIG. 3.

First, in step S1101, a first interpolation processing unit 107generates G component values Gr and Gb at the R and B positions in theBayer arrangement by performing interpolation based on the G1 and G2component values prior to image recovery processing. For example, Gr isthe value at the central pixel position in the region (FIG. 4B) obtainedby clipping out the G1 and G2 pixels above, below, leftward, andrightward of an R pixel 1202, which is the pixel of interest shown inFIG. 4A, and Gr can be calculated as the average value of the G1 and G2pixels 1201, 1203, 1204, and 1205. Also, Gb is the value at the centralpixel position in the region (FIG. 4C) obtained by clipping out the G1and G2 pixels above, below, leftward, and rightward of a B pixel 1206,which is the pixel of interest shown in FIG. 4A, and Gb can becalculated as the average value of the G1 and G2 pixels 1204, 1205,1207, and 1208. Note that these calculation methods are merely examples,and it is possible to use methods in which different numbers of pixelsat different positions are used in interpolation.

In step S1102, computing units 109 and 110 in FIG. 2 respectivelycalculate difference values R−Gr and B−Gb by subtracting the values ofGr and Gb calculated by the first interpolation processing unit 107 fromthe values of the R and B components at corresponding positions prior toimage recovery processing, and output the difference values to computingunits 111 and 112 respectively.

In step S1103, similarly to the first interpolation processing unit 107,a second interpolation processing unit 108 calculates G components Gr′and Gb′ at the R and B positions in the Bayer arrangement using thevalues of the G1′ and G2′ components resulting from the image recoveryprocessing, and outputs the G components Gr′ and Gb′ to the computingunits 111 and 112.

In step S1104, the computing units 111 and 112 respectively add thedifference values R−Gr and B−Gb calculated by the computing units 109and 110 to the G component values Gr′ and Gb′ calculated by the secondinterpolation processing unit 108 for the corresponding R and Bpositions. In this way, the computing units 111 and 112 generate colordifference adjustment values R″ and B″. In other words, the colordifference adjustment values R″ and B″ are generated according toEquations 1 and 2 below.

R″=R−Gr+Gr′  (1)

B″=B−Gb+Gb′  (2)

In this way, the color difference adjustment values R″ and B″ are valuesobtained by adding the differences (amounts of change) Gr′−Gr andGb′−Gb, which are the differences between the G components at the samepixel position before and after image recovery processing, to the colorcomponent values R and B prior to image recovery processing.

The follow equations are obtained based on Equations 1 and 2.

R″−Gr′=R−Gr   (3)

B″−Gb′=B−Gb   (4)

It can be understood that if the color difference adjustment values R″and B″ are used as the R and B values resulting from image recoveryprocessing, color differences equivalent to the color difference R−G andthe color difference B−G in the pixel values prior to image recoveryprocessing are maintained in the pixel values resulting from imagerecovery processing.

As described above, the image processing apparatus 100 of the presentembodiment obtains the color difference adjustment values R″ and B″ forthe R and B pixels respectively, and supplies them to the R componentoutput adjustment unit 113 and the B component output adjustment unit114 respectively. Also, if the saturation determination unit 106determines that a saturated pixel is included in the reference region,the R component output adjustment unit 113 outputs the color differenceadjustment value R″ instead of R′ output by the third filter processingunit 104, as an R component R_rec resulting from image recoveryprocessing. Also, the B component output adjustment unit 114 outputs thecolor difference adjustment value B″ instead of B′ output by the fourthfilter processing unit 105, as a B component B_rec resulting from imagerecovery processing. For this reason, even if a saturated pixel isreferred to when filter processing is performed, it is possible tomaintain the color difference between the pixel values before and afterimage recovery processing, and to suppress tinting.

Note that a configuration is possible in which, in the case where thepixel of interest (color component value) is saturated (exceeds thesaturated level), the value resulting from image recovery processing isnot replaced even if the recovery filter refers to the saturated pixel.For example, the value R′ resulting from image recovery processing maybe used as is for the pixel positions X1 and X2. In this case, if thepixel of interest is a saturated pixel, it is sufficient that thesaturation determination unit 106 does not make the determination that asaturated pixel is included (that the filter will refer to a saturatedpixel), regardless of whether or not another saturated pixel is includedin the reference region.

Next, effects of the image processing apparatus 100 of the presentembodiment will be described with reference to FIGS. 5A to 5C.

FIG. 5A is a diagram in which the pixel values of a pixel linecorresponding to an edge portion in and around a saturated portion areplotted in the X direction separately for each color component,similarly to FIG. 9B. Note that values for the R component and thevalues for the G component (Gr) at corresponding pixel positions, whichare obtained by interpolation, are shown in FIG. 5A. As was describedwith FIG. 9B, all of the values are values after white balanceprocessing, and FIG. 5A shows a state in which the G component values atthe pixel positions X1 and X2 have been lost due to saturation. Notethat in this example, it is assumed that if a value that has been lostdue to saturation is included among the G component values, the Gcomponent value after white balance processing will be the same as the Rcomponent value. Also, the broken line (R−Gr) indicates the differencevalue between the R component and the G component (Gr) (color differenceR−Gr).

FIG. 5B shows the result of performing image recovery processing on theR component and the Gr component having these values with use of therecovery filters shown in FIG. 9C that have different characteristicsfor each color component. Similarly to FIG. 9D, in the image recoveryprocessing performed on the G component, the original value cannot bereferred to due to the influence of saturation. For this reason, the Gcomponent Gr′ at the pixel positions X3 and X4, which are obtained byinterpolation based on the G components G1′ and G2′ resulting from imagerecovery processing, are higher than the R component R′ resulting fromimage recovery processing, which is not influenced by saturation, andthis causes green tinting. Note that both of the components exceed thesaturated level at the pixel positions X1 and X2, and both of them willbe reduced to the saturated level through clip processing in later-stagedevelopment processing, and therefore the difference in value will notcause tinting.

FIG. 5C, on the other hand, shows the results of image recoveryprocessing in which the recovery filters refer to the saturated pixelsX1 and X2, and in this case, the R component R′ resulting from imagerecovery processing has been replaced with the color differenceadjustment value R″ obtained by adding the color difference R—Gr priorto image recovery processing to the G component Gr′ resulting from imagerecovery processing. Note that as previously described, a configurationis possible in which, among the pixel positions X1 to X4 at which therecovery filter refers to the saturated pixels X1 and X2, replacement isnot performed with respect to the pixel positions X1 and X2 at which thepixel of interest is saturated, and instead the value R′ resulting fromimage recovery processing is used as is.

Both the difference between the values at the positions X2 and X3, andthe difference between the values at the positions X3 and X4 becomesmaller by replacing R′ with R″ at the positions X3 and X4 and thus theeffect of edge enhancement decreases. However, the color differencesR″−Gr′ at these pixel positions are equal to the color differences R−Grprior to image recovery processing, and therefore the difference betweenthe G component and the R component that arises due to the imagerecovery processing is suppressed, and thus tinting is suppressed.

Although the description has been given using the R component as arepresentative example, in the case of the B component as well, it issufficient that, similarly, the B component value at the position of apixel of interest for which the recovery filter refers to a saturatedpixel is replaced with the color difference adjustment value B″ obtainedby adding the color difference B−Gb prior to image recovery processingto the G component Gb′ resulting from image recovery processing.Accordingly, the color difference B−Gb prior to image recoveryprocessing is maintained after image recovery processing.

In this way, in the present embodiment, in and around a saturatedportion, or more specifically, in the case where a saturated pixel isreferred to in recovery filter processing, a color component value formaintaining the color difference before and after image recoveryprocessing (color difference adjustment value) at the pixel of interestis used as the color component value resulting from image recoveryprocessing. This enables suppressing tinting that occurs in and around asaturated portion due to a loss of balance between color componentsbefore and after image recovery processing that employs spatial filters.

Second Embodiment

In the first embodiment, tinting in and around a saturated portion issuppressed by switching the signal value resulting from image recoveryprocessing depending on whether or not the reference region referred toby the recovery filters includes a saturated pixel.

However, in a method in which the output signal is simply switched,there is the risk of a pseudo contour appearing in the vicinity of theboundary where the switch occurs. For this reason, a feature of theconfiguration of the present embodiment is that the appearance of apseudo contour is suppressed by performing weighted addition on thesignal value resulting from image recovery processing according to thepercentage of saturated pixels in the reference region referred to bythe recovery filters.

Since the present embodiment can also be realized by an image processingapparatus having the functional configuration that was described in thefirst embodiment with reference to FIG. 2, the following describesprocessing that is different from the first embodiment.

FIG. 6A is a diagram in which the R component values of a pixel linecorresponding to an edge portion in and around a saturated portion, andalso the G component (Gr) values at corresponding pixel positions thatwere obtained by interpolation are plotted in the X direction, similarlyto FIG. 5A. All of the values are values after white balance processing,and FIG. 6A shows a state in which the G component values at the pixelpositions X1 and X2 have been lost due to saturation. Also, the brokenline (R−Gr) indicates the difference value between the R component andthe G component (Gr) (color difference R−Gr). In the edge portion shownin FIG. 6A, the R component is higher than the G component due to theinfluence of diffraction, and therefore tinting has occurred.

FIG. 6B shows the result of performing image recovery processing on theR component and the Gr component having these values with use of therecovery filters shown in FIG. 9C that have different characteristicsfor each color component. Similarly to FIG. 5B, the G component Gr′ atthe pixel positions X3 and X4 has a higher value than the R component R′resulting from image recovery processing, which is not influenced bysaturation, and this causes green tinting.

FIG. 6C shows results in which, according to the method of the firstembodiment, among the pixel positions X1 to X4 at which the recoveryfilters refer to a saturated pixel, the R component value R′ at thepixel positions X3 and X4 at which the pixel of interest is notsaturated has been replaced with the color difference adjustment valueR″. Tinting is suppressed at these pixel positions since the colordifference before and after image recovery processing is maintained(R−Gr=R″−Gr′), but the recovery effect with respect to the R componentdecreases. Also, a step appears between the R component value at thepixel position X4 at which the output value is switched and the Rcomponent value at the pixel position X5 at which the output value isnot switched, and therefore there is the risk of a pseudo contourappearing in the image obtained after development processing.

In the present embodiment, in order to suppress the appearance of such apseudo contour, the saturation determination unit 106 obtains thepercentage of saturated pixels with respect to the total number ofpixels in the reference region (i.e., obtains a saturation rate) ratherthan determining whether or not the reference region includes asaturated pixel. Also, the R component output adjustment unit 113outputs the result of performing weighted addition on the R componentvalue R′ resulting from image recovery processing such that the weightof the color difference adjustment value R″ rises as the saturation raterises. The B component output adjustment unit 114 performs similarprocessing for the B component. The weighted addition performed by the Rcomponent output adjustment unit 113 and the B component outputadjustment unit 114 is expressed by Equations 5 and 6 below.

R _(—) rec=(1−α)×R′+α×R″  (5)

B _(—) rec=(1−α)×B′+α×B″  (6)

Here, α is the percentage of saturated pixels in the reference regionreferred to by the recovery filters corresponding to the third andfourth filter processing units 104 and 105 (i.e., is the saturationrate), and a takes a value from 0 to 1.

α=Ns/Nf   (7)

-   -   Nf: total number of pixels in reference region    -   Ns: number of saturated pixels in reference region

In the example shown in FIGS. 6A to 6D, the five-tap one-dimensionalfilter shown in FIG. 9C is used, and therefore in the case where thepixel of interest is X3, the saturated pixels that are referred to areX1 and X2, and the number of saturated pixels Ns is 2. Similarly, whenthe pixel of interest is X4, the saturated pixel that is referred to isX2, and the number of saturated pixels Ns is 1. Accordingly, thesaturation rate a with respect to the total number of pixels Nf=5 in thereference region referred to by the filters is 0.4 at the pixel positionX3, and is 0.2 at the pixel position X4.

FIG. 6D shows the signal values R′_rec resulting from image recoveryprocessing in which the R component values at the pixel positions X3 andX4 are weighted addition values reflecting R′. As shown in this figure,the step from the pixel position X4 to the pixel position X5 has beenreduced in the R signal resulting from image recovery processing. At thepixel positions X2 and X3 as well, the balance between the colorcomponents is closer to the balance prior to image recovery processingthan in the results of performing normal image recovery processing (FIG.6B), and the obtained recovery effect is better than in the results ofemploying the method of the first embodiment (FIG. 6C).

In this way, in the present embodiment, in the case where a saturatedpixel is referred to in recovery filter processing, a color differenceadjustment value in which the signal value resulting from image recoveryprocessing is reflected according to the percentage of saturated pixelsin the reference region referred to by the recovery filters, is used asthe color component value resulting from image recovery processing. Thismakes it possible to realize an effect of suppressing tinting in andaround a saturated portion while also suppressing the appearance of apseudo contour that appears in the case where color differenceadjustment values are used, as is, as color component values resultingfrom image recovery processing. Also, the effect of the pixel imagerecovery processing can be improved since the color component that isnot included in the color difference adjustment value and results fromimage recovery processing is reflected.

Other Embodiments

Note that the above embodiments describe configurations in which the Rcomponent and the B component are corrected based on the premise thatamong the color components that pass through the primary color filter ofa current ordinary imaging element, the sensitivity to the G componentis the highest, and saturation most readily occurs with the G component.In particular, the first embodiment describes a configuration in which,in the case where the recovery filters refer to a saturated pixel, imagerecovery processing is applied only to the luminance component (Gcomponent), and the R and B components are adjusted to values thatmaintain the color difference with the G component prior to imagerecovery processing.

However, the basic technical idea of the present invention is thecorrection of a portion of color components resulting from imagerecovery processing so as to prevent the loss of balance between thecolor components (difference between color component values) before andafter image recovery processing. Accordingly, the present invention isnot necessarily limited to correcting the R component and the Bcomponent. For example, by replacing the G component in the abovedescription with another color component that readily becomes saturated,it is possible to apply the present invention to recovery processingperformed on an image obtained by an imaging element that has differentsensitivity characteristics or an imaging element having a color filterthat is made up of color components other than RGB or has colorcomponents other than RGB. Also, the effects of the present inventioncan be obtained even if the color differences before and after recoveryprocessing do not exactly match each other, as long as the changebetween before and after recovery processing is within a predeterminedrange.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-174967, filed on Aug. 26, 2013, which is hereby incorporated byreference herein its entirety.

What is claimed is:
 1. An image processing apparatus comprising: afilter unit configured to perform image recovery processing by applyinga spatial filter for each of color components of an image; adetermination unit configured to determine whether or not a saturatedpixel is included in a reference region that is a region of pixelsreferred to by the spatial filter; and an adjustment unit configured toadjust a color component among the color components resulting from theimage recovery processing according to a determination result obtainedby the determination unit.
 2. The image processing apparatus accordingto claim 1, wherein in a case where the determination unit determinedthat a saturated pixel is included in the reference region, theadjustment unit adjusts the color component resulting from the imagerecovery processing with use of the color component prior to the imagerecovery processing.
 3. The image processing apparatus according toclaim 1, wherein in a case where the determination unit determined thata saturated pixel is included in the reference region, the adjustmentunit adjusts the color component resulting from the image recoveryprocessing with use of a value adjusted such that a color differencebetween the color components is maintained before and after the imagerecovery processing.
 4. The image processing apparatus according toclaim 3, wherein the color difference is calculated for another colorcomponent not included at a pixel position where the image recoveryprocessing is applied, by performing interpolation based on values ofthe other color component in the periphery of the pixel position.
 5. Theimage processing apparatus according to claim 3, wherein the colorcomponents include an R component, a G component, and a B component, andthe adjustment unit adjusts the R component and the B component suchthat a color difference with the G component is maintained before andafter the image recovery processing.
 6. The image processing apparatusaccording to claim 3, wherein in a case where the determination unitdetermined that a saturated pixel is included in the reference region,the adjustment unit performs the adjustment by replacing the value ofthe color component resulting from the image recovery processing withthe adjusted value.
 7. The image processing apparatus according to claim3, wherein in a case where the determination unit determined that asaturated pixel is included in the reference region, the adjustment unitperforms the adjustment by performing weighted addition on the value ofthe color component resulting from the image recovery processing and theadjusted value.
 8. The image processing apparatus according to claim 7,wherein when performing the weighted addition, the adjustment unit givesa higher weight to the value of the color component resulting from theimage recovery processing as the percentage of saturated pixels in thereference region increases.
 9. The image processing apparatus accordingto claim 1, wherein the adjustment unit does not perform the adjustmentin a case where a color component value is saturated at a pixel positionwhere the image recovery processing is applied.
 10. The image processingapparatus according to claim 1, wherein the determination unitdetermines that a pixel is the saturated pixel if the value of the pixelis greater than or equal to a predetermined threshold value.
 11. Animage processing apparatus comprising: a filter unit configured toperform image recovery processing by applying a spatial filter for eachof color components of an image; a determination unit configured todetermine whether or not a pixel whose value is greater than or equal toa predetermined threshold value is included in a reference region thatis a region of pixels referred to by the spatial filters; and anadjustment unit configured to adjust a color component among the colorcomponents resulting from the image recovery processing according to adetermination result obtained by the determination unit.
 12. A method ofcontrolling an image processing apparatus, comprising: a filter step ofperforming image recovery processing by applying a spatial filter foreach of color components of an image; a determination step ofdetermining whether or not a saturated pixel is included in a referenceregion that is a region of pixels referred to by the spatial filters;and an adjustment step of adjusting a color component among the colorcomponents resulting from the image recovery processing according to adetermination result obtained in the determination step.
 13. A method ofcontrolling an image processing apparatus, comprising: a filter step ofperforming image recovery processing by applying a spatial filter foreach of color components of an image; a determination step ofdetermining whether or not a pixel whose value is greater than or equalto a predetermined threshold value is included in a reference regionthat is a region of pixels referred to by the spatial filters; and anadjustment step of adjusting a color component among the colorcomponents resulting from the image recovery processing according to adetermination result obtained in the determination step.
 14. Anon-transitory computer-readable storage medium that stores a programthat causes a computer to function as the image processing apparatusaccording to claim
 1. 15. A non-transitory computer-readable storagemedium that stores a program that causes a computer to function as theimage processing apparatus according to claim 11.